Add auto-evals retrieval and analysis (early per-step progress signal)#39
Open
AliciaFrame wants to merge 2 commits into
Open
Add auto-evals retrieval and analysis (early per-step progress signal)#39AliciaFrame wants to merge 2 commits into
AliciaFrame wants to merge 2 commits into
Conversation
…ep evals Foundry attaches an OpenAI Evals object to every fine-tuning job and runs it every eval_interval steps (default 5). The per-step grader pass-rate curve is an early signal of whether a job is learning, well before the final model exists. - scripts/analyze_auto_evals.py: prints the per-step pass-rate curve with trend diagnosis, and dumps per-sample grader results for any step to JSONL. Uses common.get_clients; small output_items page size to avoid the API timeout. - references/auto-evals.md: object model, API/SDK usage, curve interpretation, and the large-page timeout gotcha. - SKILL.md / README.md: register the script and reference; add the timeout to Common Errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds analyze_auto_evals.py to retrieve and analyze the auto-generated evals Foundry attaches to every fine-tuning job. Foundry runs a grader over the validation set every eval_interval steps (default 5); the per-step pass-rate curve is an early signal of whether the job is learning, before the final model exists. Includes references/auto-evals.md and SKILL.md/README.md registration, plus the output_items large-page timeout gotcha (use limit<=20). Validated live on an RFT job: pass rate 1.1 percent (step 0) to 23.5 percent (step 115) across 23 runs.